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A  Suite  of  Tools  for  Technology  Assessment 


What  is  Technology? 

“Technology  is  defined  as  the  practical  application  of 
knowledge  to  create  the  capability  to  do  something  entirely  new 
or  in  an  entirely  new  way.  This  can  be  contrasted  to  scientific 
research,  which  encompasses  the  discovery  of  new  knowledge 
from  which  new  technology  is  derived,  and  engineering  which 
uses  technology  derived  from  this  knowledge  to  solve  specific 
technical  problems.” 


-  NASA  Technology  Plan 
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What  is  Technology? 

“l.a.  The  application  of  science,  especially  to  industrial  or  commercial 
objectives*,  b.  The  entire  body  of  methods  and  materials  used  to 
achieve  such  objectives.” 


-  The  American  Heritage  Dictionary 

Or  in  NASA’s  case  sjwce 

•Technology  development  lies  within  the  context  of  part  a.  and  as  such 
is  the  subject  of  the  remainder  of  this  presentation. 

•Engineering  makes  use  of  technology  within  the  context  of  part  b.  In 
this  context,  technology  may  be  “old  (passe),”  “off-the-shelf 
(commercially  available),”  or  new  (at  various  levels  of  maturity 
{TRLs}) 


A  Suite  of  Tools  for  Technology  Assessment 
What  is  Technology? 

Technology  development  can  best  be  distinguished  from 
engineering  development  in  that  it  requires  venturing  into  the 
realm  of  unknowns  -  beyond  the  ability  of  individuals  to  make 
informed  judgements  based  on  their  experience,  i.e.- 

HC  SVNT  DRACONES  * 


The  Lenox  Globe  (ca.  1503-07), 


A  Suite  of  Tools  for  Technology  Assessment 
Why  do  Technology  Assessment? 


You  don’t  know  what  you  don’t  know! 

Failure  to  accurately  assess  technology  requirements 
contributes  significantly  to  schedule  slip  and  cost 
overrun. 

Even  “heritage”  systems  can  require  technology 
development  when  they  are  incorporated  into  a  new 
architecture  with  different  operational  environments. 

Technology  development  cannot  be  done  to  schedule  - 
breakthroughs  are  rarely  performed  “on  demand”. 

Having  a  “marching  army”  in  place,  the  middle  of  a 
development  program  is  no  place  to  be  relying  on 
“miracles”  occurring  on  schedule! 
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The  Beginning  of  TRL’s 

•  The  idea  of  ascribing  levels  of  maturity  to  technology  was  first 
documented  in  a  paper  “THE  NASA  TECHNOLOGY  PUSH  TOWARDS 
FUTURE  SPACE  MISSION  SYSTEMS,  ”  (Saden,  Povinelli  &  Rosen, 
1989). 

•  This  was  a  significant  change  in  emphasis  on  the  part  of  NASA, 
where  technology  had  previously  viewed  as  merely  having  a 
supporting  role. 

•  The  change  in  role  was  the  result  of  a  revision  in  the  National  Space 
Policy  stating  that  NASA’s  technology  program  “-shares  the  mantle 
of  responsibility  for  shaping  the  Agency's  future-.” 
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Process  Improvements  and 
Management  of  Technical  Uncertainty 
Sufficient  to  Cut  Non-Recurring  Costs 


YEARS 


YEARS 


Pratt-Whitney  Rocketdyne 
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What  does  Technology  Impact? 

•  Stakeholder  Expectation:  GAO  studies  have  consistently  identified  the 
“mismatch”  between  stakeholder  expectation  and  developer  resources 
(specifically  the  resources  required  to  develop  the  technology  necessary  to 
meet  program/project  requirements)  as  a  major  driver  in  schedule  slip  and 
cost  overrun. 

•  Requirements  Definition:  If  requirements  are  defined  without  fully 
understanding  the  resources  required  to  accomplish  needed  technology 
developments  then  the  program/project  is  at  risk.  Technology  assessment 
must  be  done  iteratively  until  requirements  and  available  resources  are 
aligned  within  an  acceptable  risk  posture. 

•  Design  Solution:  As  in  the  case  of  requirements  development,  the  design 
solution  must  iterate  with  the  technology  assessment  process  to  ensure  that 
performance  requirements  can  be  met  with  a  design  that  can  be 
implemented  within  the  cost,  schedule  and  risk  constraints. 
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What  does  Technology  Impact? 

•  Risk  Management:  In  many  respects,  technology  assessment  can 
be  considered  a  subset  of  risk  management  and  as  such  should  be 
a  primary  component  of  the  risk  assessment. 

•  Technical  Assessment:  Technology  assessment  is  also  a  subset  of 
technical  assessment  and  implementing  the  assessment  process 
provides  a  substantial  contribution  to  overall  technical  assessment. 

•  Trade  Studies:  Technology  assessment  is  a  vital  part  of  determining 
the  overall  outcome  of  Trade  Studies,  particularly  with  decisions 
regarding  the  use  of  heritage  equipment. 
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What  does  Technology  Impact? 

•  Verification/Validation:  The  verification/validation  process  needs  to 
incorporate  the  requirements  for  technology  maturity  assessment  in 
that  in  the  end  maturity  is  demonstrated  only  through  test  and/or 
operation  in  the  appropriate  environment. 

•  Lessons  Learned:  Part  of  the  reason  for  the  lack  of  understanding 
of  the  impact  of  technology  on  programs/projects  is  that  we  have  not 
systematically  undertaken  the  processes  to  understand  impacts. 
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So  -  How  do  you  do  Technology  Assessment? 

•  It  is  a  two-step  process  that  involves: 

-  The  determination  of  the  Technology  Readiness  Levels  (TRLs)  (i.e. 
current  level  of  maturity)  of  all  of  the  systems,  subsystems  and 
components  required  to  meet  program/project  requirements. 

-  The  determination  of  the  Advancement  Degree  of  Difficulty  (AD2) 
(i.e.,  what  is  required  to  advance  the  immature  technologies  from  their 
current  TRL  to  a  level  that  permits  infusion  into  the  program/project 
within  cost,  schedule  and  risk  constraints. 
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What  is  a  TRL? 


•  At  its  most  basic,  it  is  maturity  defined  at  a  given  point  in 
time  by  what  has  been  done  and  under  what  conditions. 
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TRL  Descriptions 


Technology 
Readiness 
Level  -  (TRL) 

Definition 

Hardware  Description 

Software  Description 

Exit  Criteria 

1 

Basic  principles  observed 
and  reported 

Scientific  knowledge 
generated  underpinning 
hardware  technology 
concepts/applications. 

Scientific  knowledge 
generated  underpinning  basic 
properties  of  software 
architecture  and 
mathematical  formulation. 

Peer  reviewed  publication  of 
research  underlying  the 
proposed  concept/application 

2 

Technology  concept  or 
application  formulated 

Invention  begins,  practical 
application  is  identified  but  is 
speculative,  no  experimental 
proof  or  detailed  analysis  is 
available  to  support  the 
conjecture. 

Practical  application  is 
identified  but  is  speculative, 
no  experimental  proof  or 
detailed  analysis  is  available 
to  support  the  conjecture. 
Basic  properties  of 
algorithms,  representations  & 
concepts  defined.  Basic 
principles  coded. 

Experiments  performed  with 
synthetic  data. 

Documented  description  of 
the  application/concept  that 
addresses  feasibility  and 
benefit 

3 

Analytical  and/or 
experimental  critical 
function  or  characteristic 
proof-of-concept 

Analytical  studies  place  the 
technology  in  an  appropriate 
context  and  laboratory 
demonstrations,  modeling  and 
simulation  validate  analytical 
prediction. 

Development  of  limited 
functionality  to  validate  critical 
properties  and  predictions 
using  non-integrated  software 
components 

Documented 

analytical/experimental  results 
validating  predicitions  of  key 
parameters 
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What  is  AD2? 

•  The  TRL  is  just  one  part  of  the  equation  -  and  the  initial 
assessment  establishes  the  baseline  for  the 
program/project. 

•  The  more  fundamental  question  is  what  is  required  (in 
terms  of  cost,  schedule  and  risk  to  move  the  technology 
from  where  it  is  to  where  it  needs  to  be. 
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What  is  an  Advancement  Degree  of  Difficulty  (AD2)? 

•  AD2  is  a  method  of  dealing  with  the  other  aspects  beyond  TRL,  it  is 
the  description  of  what  is  required  to  move  a  system,  subsystem  or 
component  from  one  TRL  to  another. 

•  It  takes  into  account: 

-  Design  Readiness  Level 

-  Manufacturing  Readiness  Level  (MRL) 

-  Integration  Readiness  Level  (IRL) 

-  Software  Readiness  Level  (SRL) 

-  Operational  Readiness  Level 

-  Human  Readiness  Levels  (HRL)  (skills) 

-  Capability  Readiness  Levels  (CRL)  (people  and  tools) 

-  organizational  aspects  (ability  of  an  organization  to  reproduce  existing 
technology) 

-  Etc. 
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What  is  an  Advancement  Degree  of  Difficulty  (AD2)? 


Degree 

of 

Difficulty 

Description 

9 

100%  Development  Risk  -  Requires  new  development  outside  of  any  existing  experience  base.  No  viable 
approaches  exist  that  can  be  pursued  with  any  degree  of  confidence.  Basic  research  in  key  areas  needed 
before  feasible  approaches  can  be  defined. 

8 

80%  Development  Risk  -  Requires  new  development  where  similarity  to  existing  experience  base  can  be 
defined  only  in  the  broadest  sense.  Multiple  development  routes  must  be  pursued. 

7 

60%  Development  Risk  -  Requires  new  development  but  similarity  to  existing  experience  is  sufficient  to 
warrant  comparison  in  only  a  subset  of  critical  areas.  Multiple  development  routes  must  be  pursued 

6 

50%  Development  Risk  -  Requires  new  development  but  similarity  to  existing  experience  is  sufficient  to 
warrant  comparison  in  only  a  subset  of  critical  areas.  Dual  development  approaches  should  be  pursued  in 
order  to  achieve  a  moderate  degree  of  confidence  for  success.  (Desired  performance  can  be  achieved  in 
subsequent  block  upgrades  with  high  degree  of  confidence.) 

5 

40%  Development  Risk  -  Requires  new  development  but  similarity  to  existing  experience  is  sufficient  to 
warrant  comparison  in  all  critical  areas.  Dual  development  approaches  should  be  pursued  to  provide  a  high 
degree  of  confidence  for  success. 

4 

30%  Development  Risk  -  Requires  new  development  but  similarity  to  existing  experience  is  sufficient  to 
warrant  comparison  across  the  board.  A  single  development  approach  can  be  taken  with  a  high  degree  of 
confidence  for  success. 

3 

20%  Development  Risk  -  Requires  new  development  well  within  the  experience  base.  A  single  development 
approach  is  adequate. 

2 

10%  Development  Risk  -  Exists  but  requires  major  modifications.  A  single  development  approach  is 
adequate. 

1 

0%  Development  Risk  -  Exists  with  no  or  only  minor  modifications  being  required.  A  single  development 
approach  is  adequate. 
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When  do  you  do  a  Technology  Assessment  (TA)? 

•  It  is  extremely  important  that  a  Technology  Assessment  process  be 
defined  at  the  beginning  of  the  program/project. 

•  It  is  also  extremely  important  that  it  be  performed  at  the  earliest 
possible  stage  (concept  development)  and  repeated  periodically 
throughout  the  program/project  through  PDR. 

•  Inputs  to  the  process  will  vary  in  level  of  detail  according  to  the 
phase  of  the  program/project  in  which  it  is  conducted. 

•  Even  though  there  is  a  lack  of  detail  in  pre-phase  A,  the  TA  will  drive 
out  the  major  critical  technological  advancements  required. 
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How  do  you  start? 


Define  Your  Terms! 
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Develop  a  Work  Breakdown  Structure  (WBS) 


System  (1 .0) 

i - 1 - 1 

Subsystem  A  (1.1)  Subsystem  B  (1.2)  Subsystem  C  (1.3) 


i  i 

Component  a  (1.2.1)  Component  (3  (1.2.2)  Component  5  (1.2.3) 
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Start  the  Process 
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Architectural  Study  &  Technology  Assessment  Interaction 


Requirements 

X 

— ► 

Concepts 

Architecture  Studies 

t  i  t  i 


■>|  System  Design 

m 


TRL/AD2  Assessment 

i  1 1  t  i 


i 


i 


Technology  Maturation 
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Summary 

•  The  TRL  assessment  provides  the  baseline  maturity  at  the  start 
of  a  program/project  and  is  the  basis  for  the  preparation  of  the 
Technology  Readiness  Assessment  Report  required  for 
delivery  at  PDR. 

•  The  AD2  assessment  provides  the  basis  for  the  development  of 
the  Technology  Development  Plan  and  for  improved  accuracy 
of  the  development  of  program/project  cost,  schedule  and  risk. 

•  Technology  Assessment  is  vital  to  program/project  success! 
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Helpful  Tools 

•TRL  Calculator 
•AD2  Calculator 
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